Visual Fingerprinting

Unlike the audio fingerprinting algorithm, researchers have not yet developed a common method by which gestures can be recognized. For our visual fingerprinting, we use a machine learning classifier called a "random forest" and features derived from data that a kinect sensor provides. One of the biggest problems with using the kinect for gesture recognition is the fact that people are all different shapes and sizes. An adult is much larger than a child, and because of this, a classifier trained on data obtained from studying adults will fail at recognizing what children do. To prevent this we save joint angles, calculated from the geometry of the upper body, as defining features to our random forest. The angles that people make when posing for a gesture tend to be the same regardless of the body type and size. This means that we can build a classifier that works just as well on small children as adults.

Audio Fingerprinting

If you’ve ever used the Shazam app, you may already be familiar with audio fingerprinting. Audio fingerprinting is the process of analyzing a song or audio clip to determine what it is. First, an audio clip is sampled and broken down into the different frequencies that comprise it. From this, a unique fingerprint hash is created. This can be thought of as a type of numerical fingerprint, hence the name audio fingerprinting. Once the fingerprint is created, an algorithm checks it against a database of all of the other recorded audio fingerprints. If the algorithm finds a match or partial match, it is said to have recognized the song. The computer can then determine what the song is, and where in the song the sample can be found. We use this method to first recognize the song that is playing, and once the song has been identified, we use a predefined dance script to have HUBO dance.